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Real Time Event Strategies 


• Client Polling 

Easy to implement 

Really inefficient - 99+% noop 

• HTTP Long Poll 

Keep HTTP socket open, block on requests 
Heavy server resource usage 

• Websockets 

Connected sockets in web browser 
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Webhooks 

Register callback URL 

Need highly available "catcher" service 

Only available for service owners 

MQTT / AMQP 

Optimized pub/sub systems 
Web client support requires bridging 




What is MQTT? 

• MQTT history 



Created in 1999 by IBM & Cirrus Link, OASIS standard since 2013 
Publish / Subscribe paradigm that requires a message broker 
Designed for a "small code footprint" and "limited network bandwidth" 


• Cloud IoT Services based on MQTT 

Google IoT Core - https://cloud.google.com/iot-core/ 

IBM Watson IoT - https://www.ibm.com/internet-of-things/ 

Amazon IoT - https://aws.amazon.com/iot-platform 

Microsoft Azure IoT - https://azure.microsoft.com/en-us/services/iot-hub/ 

• Open Source Servers / Clients everywhere 

- Notable-ESP8266 

• PubSubClient - Arduino MQTT implementation 
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What is Pub / Sub 


• Design Pattern 

• Common message bus 

• Everyone can publish to it 

• Messages directed to topics 

• Consumers subscribe to specific topics 
(possibly by wildcard) 

• Great architecture for handling many to 
many interactions 
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Mosquitto 


• Open Source by Eclipse Foundation 

• Written in C / highly performant 

• Packaged for most Linux distros 

• Includes CLI tools for pub/sub 

• https://mosquitto.org/ 
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MQTT Message Format 


QoS Retain Topic Payload 


0,1,2 

0,1 

220 

characters 

Any content, up to 2 GB 


• QoS - 0 best effort, 1 deliver at least once, 2 deliver exactly once 

• Retain - content will be stored on the server, replayed on connect, defaults 
to not stored 

• Topic - name for message, / are special 

• Payload - anything, 2 GB payload limit 

Note: no metadata on packets (like time sent), must put it in payload manually 
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Example sensible topic schema 


{app name}/{location}/{device type}/{sensor type}/{reading type} 


Allows the following subscriptions: 

{app name}/{location}/# - see everything at a location 
{app name}/+/+/{sensor type}/+ - see all particulate sensors 
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MQTT Wills 


• Event based system - messages only sent when event happens 

• If nothing is sent, is the client healthy with no new data, or did it disappear? 

• Clients ca set a "Will" on client connect 

a message stored in the server that will be sent if the socket connection to the client breaks 
building block for fault tolerance 
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Example Applications 
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Home Weather Station 



Bill of Materials 

• Raspberry Pi 3 

• RTL-SDR 

• Oregon Scientific Sensors 

• http://github.com/sdague/arwn 
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mqtt: 

broker: 10.42.0.3 

sensor 1: 
platform: arwn 


(jjtffjj) mosefuitto ▼ 



arwn/temperature/Freezer {"bat": "LOW", "sensor_id": "6a:03", "humid": 

53.0, "temp": 

-10.8, "dewpoint": 

-23.2, 

"units": "F", "timestamp": 

1527695510} 

> 


arwn/temperature/Arwen Room {"bat": "OK", "sensor_id": "ce:08", "humid 

: 54.0, "temp" 

72.7, "dewpoint" 

55.1, 

"units": "F", "timestamp": 

1527695511} 



arwn/wind {"bat": "OK", "sensor_id": "33:00", "timestamp": 1527695512, 

"units": "mph" 

"gust": 0.9, "speed": 2 

5, "direction": 315.0} 
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Home Assistant - Python 3 home automation 


Home Assistant 


Home 
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Developer tools 
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Rain Gauge - Retain Topics 


arwn/totals/rain 

arwn/rain 

arwn/rain/today 

arwn/rain 

arwn/rain/today 

arwn/rain 

arwn/rain/today 

arwn/rain 

arwn/rain/today 


{"timestamp": 1528084804, "total": 70.78} 
{"timestamp": 1528084868, "total": 70.78} 
{"timestamp": 1528084868, "since_midnight": 0.00} 
{"timestamp": 1528091737, "total": 70.818} 
{"timestamp": 1528091737, "since_midnight": 0.04} 
{"timestamp": 1528092583, "total": 70.944} 
{"timestamp": 1528092583, "since_midnight": 0.16} 
{"timestamp": 1528107858, "total": 71.358} 
{"timestamp": 1528107858, "since_midnight": 0.58} 


arwn/rain {"timestamp": 1528171098, "total": 71.358} 

{"timestamp": 1528171098, "since_midnight": 0.58} 

... rollover event 


arwn/rain {"timestamp": 1528171218, "total": 71.358} 

arwn/totals/rain {"timestamp": 1528171218, "total": 71.358} 

{"timestamp": 1528171218, "since_midnight": 0.0} 
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Self emptying bucket 

• 3 increments added based on time between 
bucket dumps 

Reports Rain Total 

• (10 th s of mm accumulator) 

Wunderground API 

• rainin - [rain inches over the past hour)] — the 
accumulated rainfall in the past 60 min 

• dailyrainin - [rain inches so far today in local 
time] 




When Should I Charge My Car? 



Supports Time of Departure Charging 

Be fully charged by a set time every day 

Time of Use metering at our home 

Peak is 2 - 7pm Weekdays 

Peak power costs 120%, off peak costs 89% 

What's the difference in power off peak? 

What is generating the power? 

What's the carbon intensity at different times? 

Can we make data available in real time? 

Others might want to do things with this data 


n || "ii 
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ny-power microservices architecture 


NY ISO CSVs 
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ny-power-archive 


MQTT publish 


t 
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ny-power/status/fuel-mix/updated {"ts": "05/09/2018 00:05:00"} 

ny-power/upstream/fuel-mix/Nuclear {"units": "MW", "value": 4114, "ts": "05/09/2018 00:05:00"} 
ny-power/upstream/fuel-mix/Dual Fuel {"units": "MW", "value": 1400, "ts": "05/09/2018 00:05:00"} 
ny-power/upstream/fuel-mix/Natural Gas {"units": "MW", "value": 2144, "ts": "05/09/2018 00:05:00"} 


ny-power/archive/co2/24h {"units": "g/ kWh", "values": [162.698, 
163.928,161.587 ... ], "ts": [...] 


(jMj) mosGtuitto 


ny-power/computed/co2 {"units": "g / kWh", "value": 135.088, "ts": "05/09/2018 00:05:00"} 






ny-power-mqtt (access with "mosquitto_sub -h 169.60.78.157 -t ny-power/# -v") 
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ny-power topics 


{app name}/{source}/{details}/{more details} 


Allows the following subscriptions: 

ny-power/upstream/fuel-mix/{fuel type} 

ny-power/computed/co2 

ny-power/archive/co2/24h 


@sdague 


var client = new Paho.MQTT.Client("mqtt.ny-power .org" , Number("80") , "client-" + Math.randomQ) ; 


// set callback handlers 

client.onMessageArrived = onMessageArrived; 

// connect the client 

client.reconnect = true; 

client.connect({onSuccess: onConnect}); 

// called when the client connects _ 

function onConnect() { 

// Once a connection has been made, make a subscription and send a message. 

console. logC'onConnect") ; 
client.subscribe ("ny- power/computed/co2") ; 
client.subscribe ("ny-power/archive/co2/24h") ; 
client.subscribe( "ny-power/application/webui" ); 
client.subscribe ("ny-power/upstream/fuel-mix/#" ); 

} 


// called when a message arrives 

function onMessageArrived(message) { 

console.log( "onMessageArrived: "+message.destinationName +. message.payloadString); 

if (message.destinationName == "ny-power/computed/co2") { 
var data = JSON.parse(message.payloadString); 

$( "#co2-per-kwh" ).html (Math .round(data.value)); 

$("#co2-units") .html(data.units); 

$("#co2-updated") .html(data.ts); 

I_ 1 _ 

if (message.destinationName.startsWith( "ny-power/upstream/fuel-mix" )) { 
fuel_mix_graph(message); 

} 

if (message.destinationName == "ny-power/archive/co2/24h") { 

(CpSQ3gU6 var data = JSON.parse(message.payloadString); 









The current NY ISO Grid C02 / kWh 
http://ny-pOWer.org Current 163 g / kWh 


C02 calculated from 2016 totals 
(MW & Emissions Per fuel 
source) 

Provided as MQTT stream 

Answer: complete charging by 
5am before load / NG starts 
ramping up 



http://github.com/IBM/ny-power 

• Helm Kube application 

• Core logic in python 

• 5 pods 


Current Fuel Mix on NY ISO Grid 



Natural Gas 



Other Renewables 


Other Fossil Fuels 
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Put MQTT in your toolkit 


• In a world awash in data, efficient event streams are critical 

• Open event streams can be a public good 

• MQTT has natural growth in the IoT space 

• The Pub / Sub programming makes you think of problems in new ways 


Thank You! 

Twitter: @sdague 

Email: sean.dague@ibm.com / sean@dague.net 

Blog: https://dague.net - software engineering, open source projects, climate & energy 
Get the code: 

• arwn: https://dague.net/arwn 

• ny-power project: https://dague.net/ny-power 
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CALL FOR CODE 

COMMIT TO THE CAUSE. PUSH FOR CHANGE. 

callforcode.org 


Call for Code inspires 
developers to solve pressing 
global problems with 
sustainable software 
solutions, delivering 
on their vast potential to do 
good. 

Bringing together NGOs, 
academic institutions, 
enterprises, and startup 
developers to compete build 
effective disaster mitigation 
solutions, with a focus on 
health and well-being. 

The American Red Cross, 
and the United Nations 
Human Rights Office 
combine for the Call for Code 
Award to elevate the profile 
of developers. 


Award winners will receive long¬ 
term support through the Linux 
Foundation, financial prizes, 
the opportunity to present their 
solution to leading VCs, and will 
deploy their solution through 
IBM’s Corporate Service Corps. 
Developers will jump-start their 
project with dedicated IBM Code 
Patterns, combined with 
optional enterprise technology 
to build projects over the course 
of three months. 

Judged by the world’s most 
renowned technologists, the 
grand prize will be presented in 
October at an Award Event. 



